<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML
><HEAD
><TITLE
>Storing data</TITLE
><META
NAME="GENERATOR"
CONTENT="Modular DocBook HTML Stylesheet Version 1.79"><LINK
REL="HOME"
TITLE="DataparkSearch Engine 4.53"
HREF="index.en.html"><LINK
REL="PREVIOUS"
TITLE="Sub-documents"
HREF="dpsearch-subdocs.en.html"><LINK
REL="NEXT"
TITLE="Cache mode storage"
HREF="dpsearch-cachemode.en.html"><LINK
REL="STYLESHEET"
TYPE="text/css"
HREF="datapark.css"><META
NAME="Description"
CONTENT="DataparkSearch - Full Featured Web site Open Source Search Engine Software over the Internet and Intranet Web Sites Based on SQL Database. It is a Free search software covered by GNU license."><META
NAME="Keywords"
CONTENT="shareware, freeware, download, internet, unix, utilities, search engine, text retrieval, knowledge retrieval, text search, information retrieval, database search, mining, intranet, webserver, index, spider, filesearch, meta, free, open source, full-text, udmsearch, website, find, opensource, search, searching, software, udmsearch, engine, indexing, system, web, ftp, http, cgi, php, SQL, MySQL, database, php3, FreeBSD, Linux, Unix, DataparkSearch, MacOS X, Mac OS X, Windows, 2000, NT, 95, 98, GNU, GPL, url, grabbing"></HEAD
><BODY
CLASS="CHAPTER"
BGCOLOR="#FFFFFF"
TEXT="#000000"
LINK="#0000C4"
VLINK="#1200B2"
ALINK="#C40000"
><!--#include virtual="body-before.html"--><DIV
CLASS="NAVHEADER"
><TABLE
SUMMARY="Header navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TH
COLSPAN="3"
ALIGN="center"
>DataparkSearch Engine 4.53: Reference manual</TH
></TR
><TR
><TD
WIDTH="10%"
ALIGN="left"
VALIGN="bottom"
><A
HREF="dpsearch-subdocs.en.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="80%"
ALIGN="center"
VALIGN="bottom"
></TD
><TD
WIDTH="10%"
ALIGN="right"
VALIGN="bottom"
><A
HREF="dpsearch-cachemode.en.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
></TABLE
><HR
ALIGN="LEFT"
WIDTH="100%"></DIV
><DIV
CLASS="CHAPTER"
><H1
><A
NAME="HOWSTORE"
></A
>Chapter 5. Storing data</H1
><DIV
CLASS="TOC"
><DL
><DT
><B
>Table of Contents</B
></DT
><DT
>5.1. <A
HREF="dpsearch-howstore.en.html#SQL-STOR"
>SQL storage types</A
></DT
><DT
>5.2. <A
HREF="dpsearch-cachemode.en.html"
>Cache mode storage</A
></DT
><DT
>5.3. <A
HREF="dpsearch-perf.en.html"
><SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
> performance issues</A
></DT
><DT
>5.4. <A
HREF="dpsearch-searchd.en.html"
>SearchD support</A
></DT
><DT
>5.5. <A
HREF="dpsearch-oracle.en.html"
>Oracle notes</A
></DT
></DL
></DIV
><DIV
CLASS="SECT1"
><H1
CLASS="SECT1"
><A
NAME="SQL-STOR"
>5.1. SQL storage types</A
></H1
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="SQL-STOR-GENERAL"
>5.1.1. General storage information</A
></H2
><P
><SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
> stores every words found in any defined section of
document. The count of word appearance in the document does not affect
it's weight. But the fact whether the word appears in more important
parts of the document (title, description, etc.) is taken in account however. </P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="SQL-STOR-MODES"
>5.1.2. Various modes of words storage</A
></H2
><P
>There are different modes of word storage which
are currently supported by <SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
>:
"single", "multi", "crc", "crc-multi", "cache". Default mode is "cache". Mode is
to be selected by <CODE
CLASS="OPTION"
>dbmode</CODE
> parameter of <B
CLASS="COMMAND"
>DBAddr</B
> command in both
<TT
CLASS="FILENAME"
>indexer.conf</TT
> and <TT
CLASS="FILENAME"
>search.htm</TT
>
files.</P
><PRE
CLASS="PROGRAMLISTING"
>Examples:
DBAddr mysql://localhost/search/?dbmode=single
DBAddr mysql://localhost/search/?dbmode=multi
DBAddr mysql://localhost/search/?dbmode=crc
DBAddr mysql://localhost/search/?dbmode=crc-multi</PRE
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="SQL-STOR-SINGLE"
>5.1.3. Storage mode - single</A
></H2
><A
NAME="AEN3172"
></A
><P
>When "single" is specified, all words are stored
in one table  with structure
(url_id,word,weight), where url_id is  the ID of the document which is
referenced by rec_id field in "url" table. Word has <TT
CLASS="LITERAL"
>variable
char(32)</TT
> SQL type.</P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="SQL-STOR-MULTI"
>5.1.4. Storage mode - multi</A
></H2
><A
NAME="AEN3179"
></A
><P
>If "multi" is selected, words will be located in
different 13 tables depending of their lengths. Structures of these
tables are the same with "single" mode, but fixed length char type is
used, which is usually faster in most databases. This fact makes
"multi" mode usually faster comparing with "single" mode. </P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="SQL-STOR-CRC"
>5.1.5. Storage mode - crc</A
></H2
><A
NAME="AEN3185"
></A
><P
>If "crc" mode is selected, <SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
> will
store 32 bit integer word IDs calculated by HASH32 algorithm instead of
words. This mode requires less disc space and is faster than "single"
and "multi" modes. <SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
> uses the fact that HASH32 calculates
quite unique check sums for different words. According to our tests
there are only 250 pairs of words have the same HASH32 value in the list of
about 1.600.000 unique words. Most of these pairs (&gt;90%) have at
least one misspelled word. Words information is stored in the
structure (url_id,word_id,weight), where word_id is 32 bit integer ID
calculated by HASH32 algorithm. This mode is recommended for big search
engines. </P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="SQL-STOR-CRCMULTI"
>5.1.6. Storage mode - crc-multi</A
></H2
><A
NAME="AEN3193"
></A
><P
>When "crc-multi" mode is selected, <SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
>
stores HASH32 word IDs in several tables with the same to "crc" structures depending on word lengths
like in "multi" mode. This mode usually is the most fast and
recommended for big search engines.</P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="SQL-STOR-STRUCTURE"
>5.1.7. SQL structure notes</A
></H2
><P
>Please note that we develop <SPAN
CLASS="APPLICATION"
>DataparkSearch</SPAN
> with
<SPAN
CLASS="APPLICATION"
>PostgreSQL</SPAN
> as back-end and often have no possibility to test each version
with all of other supported databases. So, if there is no table
definition in create/you_database  directory, you may found <SPAN
CLASS="APPLICATION"
>PostgreSQL</SPAN
>
definition for the same table and just adopt it for your
back-end. <SPAN
CLASS="APPLICATION"
>PostgreSQL</SPAN
> table definitions are always up-to-date.</P
></DIV
><DIV
CLASS="SECT2"
><H2
CLASS="SECT2"
><A
NAME="SQL-STOR-NONCRC"
>5.1.8. Additional features of non-CRC storage modes</A
></H2
><P
>"single" and "multi" modes support substring search. As
far as "crc" and "crc-multi" do not store words themselves and use
integer values generated by HASH32 algorithm instead, there is no
possibility of substring search in these modes.</P
></DIV
></DIV
></DIV
><DIV
CLASS="NAVFOOTER"
><HR
ALIGN="LEFT"
WIDTH="100%"><TABLE
SUMMARY="Footer navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
><A
HREF="dpsearch-subdocs.en.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="index.en.html"
ACCESSKEY="H"
>Home</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
><A
HREF="dpsearch-cachemode.en.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
>Sub-documents</TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
>&nbsp;</TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
>Cache mode storage</TD
></TR
></TABLE
></DIV
><!--#include virtual="body-after.html"--></BODY
></HTML
>